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Abstract. We analyze the semiclassical limit of spectral theory 
on manifolds whose metrics have jump-like discontinuities. Such 
systems are quite different from manifolds with smooth Riemann- 
ian metrics because the semiclassical limit does not relate to a clas- 
sical flow but rather to branching (ray-splitting) billiard dynamics. 
In order to describe this system we introduce a dynamical system 
on the space of functions on phase space. To identify the quantum 
dynamics in the semiclassical limit we compute the principal sym- 
bols of the Fourier integral operators associated to reflected and 
refracted geodesic rays and identify the relation between classical 
and quantum dynamics. In particular we prove a quantum er- 
godicity theorem for discontinuous systems. In order to do this we 
introduce a new notion of ergodicity for the ray-splitting dynamics. 



1. Introduction 

Many questions about spectra and eigenfunctions of elliptic oper- 
ators are motivated by Bohr's correspondence principle in quantum 
mechanics, asserting that a classical dynamical system manifests itself 
in the semiclassical (as Planck's constant h ^ 0) limit of its quan- 
tization. When the quantum system is given by the Laplacian A on 
the Riemannian manifold M (describing a quantum particle on M in 
the absence of electric and magnetic fields) the corresponding classi- 
cal system is the geodesic flow G* on M, so in the high energy limit 
eigenfunctions should reflect the properties of the geodesic flow. 

One of the most studied question concerns limits of eigenfunctions. 
To an eigenfunction (pj with A(f)j — Xj(f)j one can associate a mea- 
sure dfij on M with the density its phase space counterpart is a 
distribution dcuj on the unit cosphere bundle S*M, projecting to d/Xj. 
It can be deflned as follows: given a smooth function a on S*M (an 
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observable), we choose a "quantization," a pseudodifferential operator 
A — Op(a) of order zero with principal symbol a, and let 



(a, dcuj 



{A(pj)(x) (j)j{x) dx := {A(j)j, 



M 



The definition of the measures dcuj depends of course on the choice 
of the quantization map a h-> Op(a), but any two choices differ by 
an operator of a lower order, and that docs not affect the asymptotic 
behaviour of {A(f)j,(f)j). The measures duj arc sometimes called Wigner 
measures. 

A natural problem is to study the set of weak* limit points of c^^-s; it 
follows from Egorov's theorem that any limit measure is invariant under 
the geodesic flow, but the limits are quite different for manifolds with 
integrable and ergodic geodesic flows. If M has completely integrable 
geodesic flow G*, sequences of duj-s concentrate on Liouville tori in 
phase space satisfying the quantization condition. 

Let 

N{X) := #{A, < X'} 

be the the counting function of the Laplacian (as usual, we enumerate 
eigenvalues taking into account their multiplicities). Recall that, by 
the Weyl formula, 

A^(A) = A"(27r)-"n-^Vol(,S*M) + o(A"), A ^ +00 . 

If G* is ergodic, the following fundamental result (sometimes called 
quantum ergodicity theorem) holds. 

Theorem 1.1. Let M be a compact manifold with ergodic geodesic 
flow, and let A be a zero order pseudodifferential operator with principal 
symbol a a- Then 



{A(f)j,(l)j) 



a A duj 



S*M 



o(iV(A)) , A ^00, 



or, equivalently, 



1 ^ 
hm — 



{A(l)j,(f)j) - aAdu 



S*M 



where du is the normalized canonical measure on S*M . 

The theorem shows that for a subsequence of eigenfunctions of 
the full density, dcUj^ dcu, and after projecting to M we find that 
(f)^^ — )■ 1 (weak*). In other words, almost all high energy eigenfunctions 
become equidistributed on the manifold and in phase space. Various 
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versions of Theorem were proved in |CVl IHMRl IShn74l IShn93[ 
Z3] by Shnirelman, Zelditch, Colin de Verdiere and Helffer-Martinez- 
Robert, as well as in other papers. Quantum ergodicity has been es- 
tablished for billiards in \GL\ IZZj by Gerard-Leichtnam and Zelditch- 
Zworski. 



Important further questions concern the rate of convergence in ( 1.1 ), 
quantum analogues of mixing and entropy. Rudnick and Sarnak conjec- 
tured that on negatively curved manifolds, the conclusion of Theorem 



1.1 holds without averaging, or equivalently duj — )■ dcu for all eigen- 
functions; this is sometimes called quantum unique ergodicity (QUE). 
This conjecture has been proved for some arithmetic hyperbolic man- 
ifolds by Lindenstrauss, with further progress by Soundararajan and 
Holowinsky. On the other hand, A. Hassell ( |Ha] ) has shown that on 
the Bunimovich stadium billiard, there exist exceptional sequences of 
eigenfunctions concentrating on the "bouncing ball" orbits, so the ana- 
logue of the QUE conjecture does not hold for all billiards. 

The aim of this paper is to identify the correct semiclassical dy- 
namics corresponding to quantum systems with discontinuities. More 
precisely, we are looking at manifolds (possibly with boundary) whose 
metrics are allowed to have jump discontinuities across codimension 
one hypersurfaces. Such manifolds model situations in physics where 
waves propagate in matter that consists of different layers of materials. 
In the simplest case we would have two isotropic materials touching at 
a hypersurface. The metric in each layer is then given by gi = riilxYge, 
where ni{x) is the refraction index in layer i and ge is the Euclidian 
metric on the tangent bundle. Wave propagation in these media is 
described by the wave equation 

(— + A)0(x,t) = O, 

where A is the Laplace operator with respect to the metric gi and trans- 
missive boundary conditions are imposed on the solutions. The high 
energy limit of such a systems shows properties that do not remind of 
classical mechanics: singularities of solutions travel on geodesies un- 
til they hit the discontinuity. Then they are reflected and refracted 
according to the laws of geometrical optics. Consequently, there is no 
classical flow on phase space that describes the high energy limit. More- 
over, the naive generalization of Egorov's theorem fails in this situation. 
If A is a pseudodifferential operator on the manifold supported away 
from the discontinuity then the quantum mechanical time-evolution 
U{t)AU{—t) of A will in general fail to be a pseudodifferential opera- 
tor. Thus, on the algebraic level of observables the quantum-classical 
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correspondence fails. We will show that after forming an average over 
the eigenstates the quantum dynamics relates to a certain probabilis- 
tic dynamics that takes into account the different branches of geodesies 
emerging in this way. Our main result establishes a quantum ergodicity 
theorem in the case where this classical dynamics is ergodic. 

Our proof relies on a precise symbolic calculus for Fourier integral 
operators associated with canonical transformations (see Section |4]) and 
on a local Weyl law for such operators (Theorem 6.2). We construct a 
local parametrix for the wave kernel consisting of a sum of such Fourier 
integral operators (Section 5.1) and apply to them the above results. 

The usual proof of quantum ergodicity is based on the consideration 
of the positive operator obtained by squaring the average of the time- 
evolution of a pseudodifferential operator. Egorov's theorem plays an 
important role in this construction. Since it does not hold in our set- 
ting, the standard proof cannot be directly applied. Instead we use 
the local Weyl law for an operator that is not necessarily positive but 
whose expectation value with respect to any eigenfunction is positive. 
We then apply the symbolic calculus to obtain an explicit formula for 
the leading asymptotic coefficient. 

The proof of the main result is presented in full detail in Section [8j It 
should be mentioned that local Weyl laws of the type given in Theorem 



6.2 are very powerful and have many applications. In particular, a less 



explicit but similar result was recently stated and used in |TZlj and 
|TZ2j to prove quantum ergodic restriction theorems. 

Ray splitting not only occurs in the quantum systems we consider 
but also happens in situations described by systems of partial differ- 
ential equations and higher order equations. Moreover, ray-splitting 
occurs in a natural way in quantum graphs. Our results carry over in 
a straightforward manner to these situations. 

Ray-splitting billiards have been studied extensively in the Physics 
literature, see e.g. |BYNKl IBAGOPll IBAG0P21 iBltsl ICOXl [EKBI 
ITSU ITS2j and references therein. The emphasis has been on spectral 
statistics, trace formulae, eigenfunction localization ("scarring"), and 
the behaviour of periodic orbits. In the mathematical literature, the 
emphasis has been on the propagation of singularities |Ivlj and spectral 
asymptotics |Iv2l ISa2] . 

Quantum ergodicity has not previously been considered for ray- 
splitting (or branching) billiards. Our results fill an important gap in 
the semiclassical theory of such systems. It is our hope that our results 
will serve as a motivation to further study a probabilistic dynamical 
system that we introduce. 
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2. Setting: Manifolds with metric discontinuities along 

hypersurfaces 

Following Zelditch and Zworksi [ZZJ, we say that M is a compact 
manifold with piecewise smooth Lipschitz boundary if M is a compact 
subset of a smooth manifold M such that there exists a finite collection 
of smooth functions fi,--.,fi such that 

(1) rf/.l/-(0)7^0, 

(2) M has Lipschitz boundary and fr\0) f] f7\o) is an embedded 
submanifold of M. 

(3) M = {x e M : yi<j <i: fj{x) > 0}. 

A Riemannian metric on M is assumed to be the restriction of a Rie- 
mannian metric on M. We will denote the regular part of the boundary 
dM of M by dM-^^^ and the singular part by SMging. 

Suppose that X is a compact manifold with piecewise smooth Lips- 
chitz boundary and y C X is a co- dimensional one piecewise smooth 
closed hypersurface in X such that dY C dX. We will also assume that 
the completion of X\Y with respect to the inherited uniform struc- 
ture is again a manifold M with piecewise smooth Lipschitz boundary. 
Thus, cutting X open along Y results in M and we obtain a part N of 
the boundary dM with a two-fold covering map N -^Y . 

Example 2.1. In the simplest case X is oriented, Y is a closed hyper- 
surface that separates X into two parts Xi and X2. Then M will he 
the disjoint union of Xi and X2, N is the disjoint union of two copies 
ofY, and the deck transformation simply interchanges these two copies 
ofY. 

A smooth metric on M defines a metric on X\Y. Since we do not 
require that the covering map be an isometry, this metric can in 
general not be continued to a metric on X but has a jump discontinuity 
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Figure 2. Closed manifold with metric discontinuity 



at Y. Manifolds X with a metric of this form on X\Y can be thought 
of as Riemannian manifolds with metric jump discontinuities at Y. 
Note that the construction ensures that the metric can be continued 
smoothly up to Y on either side of Y in local coordinates (although 
the continuations from the left and from the right need not coincide) . 

The gluing construction defines a map T*M\n — )■ T*X\y which is 
again a two-fold covering map that lifts the original covering map. Note 
that the deck transformation of this cover does not in general preserve 
the length of covectors. 

Since has zero measure, functions in L^{M) can also be understood 
as functions in L'p{X). Moreover, functions in C{M) can be understood 
as functions in L°°(X) which are smooth away from Y and can have 
a jump discontinuity along Y . For the sake of notational simplicity, 
in the following we will not distinguish between the spaces U'(M) and 
U'{X) and understand C{M) as a subspace of L°°(X). 

We will assume that D is either the Sobolev space H^{X) or the 
space ifQ(Xint) of if ^-functions vanishing on dX. We define the Laplace 
operator A on X as the self- adjoint operator defined by the Dirichlet 
quadratic form 

(2.1) g(0,0) = / \V(l){x)fg{x)dx 

Jx\Y 

with domain D, where g is the standard Riemannian density, g{x) dx is 
the volume element and \'V(f){x) \ is the Riemannian norm of the covector 
V0. The domain of the unique self-adjoint operator generated by this 
quadratic form coincides with 

{/ e H\M)f]H\X) : (V/,n^) = -(V/,n^), f\ax = 0} 

in the case D — ifQ(Xint), and with 

{feH\M)f]H\X) : (V/,riiv) = -(V/,n^r), (V/,nax) = 0} 

in the case D = H^{X). In both formulae un and uqx are the out- 
ward pointing unit normal vector fields at A^reg and dXj-eg respectively, 
and V/ is the deck transformation of V/. Thus, in the former case 
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the operator is subject to Dirichlet boundary conditions at dX and 
transmissive boundary condition sA, N . In the latter case the operator 
is subject to Neumann boundary conditions at dX and transmissive 
boundary conditions at A^. 

Let H^{M) be the Sobolev space. Since these boundary conditions 
are eUiptic, we have 

dom(A"/2) C H'{M) 
for any s > 0. Moreover, ii s > n/2 + k then 

dom(A^/2) C C'=(M)p|C(X). 

For technical reasons, it is more convenient to consider operators 
acting in the space of half-densities on X rather than in the space of 
functions. Further on, if is a function space, we shall denote by 
'H{-,n^^'^) the corresponding space of half-densities. 

The operator g^/^ Ag^^/^ with domain 

{feL\X,n'/') : g-VVedom(A)} 

is said to be the Laplacian in the space of half- densities. Clearly, it is a 
self-adjoint operator whose spectrum coincides with the spectrum of A. 
Moreover, / is an eigenfunction of A if and only if the half-density g^^^/ 
is an eigenvector of g^/^ A g~^/^ corresponding to the same eigenvalue. 

We shall denote the Laplacian in the space of half-densities by the 
same letter A specifying, when necessary, in what space we consider 
the operator. 

3. Ray-splitting billiards 

3.1. Reflection and refraction, li x E X \ Y, let {g^^{x)} be the 
Riemannian metric on T*X and g{x,^) := J2ij=i 9^K^) d Cj- ^fxE l^eg 
then the two-fold covering map N ^ Y induces two natural metrics on 
T*X. Locally the normal bundle of Y^^g is trivial and, therefore, there 
exists a connected open neighbourhood O oi x in X such that 0\Y 
is the disjoint union of two connected components C+ and C_. After 
making such a choice, the two natural metrics {g^_l{x)} and {g^l{x)} on 
T*X are obtained by passing to the limit in (9+ and (9_ respectively. 
Let g±{x,^) be the corresponding quadratic forms, and let be the 
gi±-unit conormal vectors oriented into the g^^-sides. 

If {y,ri) e T*{X\dM), where y G X\dM and rj e T*X, let de- 
note by {x^ {y , 1]) , C,^ {y , 7])) the Hamiltonian trajectory generated by the 
Hamiltonian ^/g{x, ^) and starting at {x^{y,7]),^^{y,7])) := {y,7]). Its 
projection onto X is the geodesic emanating from the point y in the 
direction rj. Clearly, and ^* are positively homogeneous functions of 
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Tj of degree and 1 respectively. Furthermore, if {y, fj) G S*M tiien 
{x^{y,fj),^*{y,fj)) e S*M and the geodesic is parametrised by its 
length. 

The Hamiltonian trajectory (a;*,.^*) is well defined until the geodesic 
hits the boundary dM, i.e., either the set Y or the boundary of 
X. In the former case, according to the laws of geometrical optics, it 
splits into two geodesies. One of them is obtained by reflection, and 
the other is the refracted trajectory which goes through Y but changes 
its direction. In the latter case there is only the reflected trajectory. 
More precisely, there are the following possibihties. 

Assume, for the sake of definiteness, that the trajectory 



approaches Y from the g^+'Side and meets Yreg at the time t*. Let 



so that X* G Yj-cg and g+{x*,^*) = g{y,r]). Denote by the g±- 
orthogonal projections of ^* onto the cotangent space T**Y. 
Clearly, ^* = ^+ — r+ n+ where 



By definition, the reflected trajectory is the Hamiltonian trajectory 
7+(t) originating from the point (a;*,^y + r+n^) at the time t* and 
going into the (7_|_-sidc of X. If r+ = then the geodesic hits Y^-cg at 
zero angle. In this case the reflected trajectory is not well deflned. 
Assume that (7_(a;*,^y) < g+{x*,^*) and denote 



Obviously, g-{x*,^Y =t r_n^)) = g+{x*,^*). The Hamiltonian trajec- 
tory 7_(i) originating from the point (x*,^y + T_n~) at the time t* 
and going into the (y'_-sidc is called the refracted trajectory. Note that 
in this case 7-(t) is the reflection of the trajectory 7-(t) coming from 
the (7_-sidc to the point {x*,^y ~ T_n~). The corresponding refracted 
trajectory coincides with 7+(t), so that 7+(t) and 7-(t) have the same 
pair of reflected and refracted trajectories. 

If g^{x*,^Y) > g+{^*7^*) then there is no refraction. In this case one 
says that (x*,^*) is a point of total reflection. If g_{x*,^Y) = g+i^*^ C*) 
then the angle of refraction is zero and the refracted trajectory may 
not be well defined. 



7+(t) = {x\y,r]),^\y,r]j) 
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If the geodesic hits a point x* G dX-^eg then (x*,^*) is reflected in 
the same way as above. Namely, the reflected trajectory is the Hamil- 
tonian trajectory originating from {x*,^qx ~ O where ^* are 

tangential and normal components of 

3.2. Billiard trajectories. The trajectory obtained by consecutive 
reflections and/or refractions is called a billiard trajectory. In general, 
there are infinitely many billiard trajectories originating from a given 
point {y.rj) G T*{X\dM); moreover, the set of these trajectories is 
typically uncountable. We shall denote them by (x^,^*), where k is an 
index specifying the type of trajectory (see below). Each billiard tra- 
jectory (a;^,^^) consists of a collection of geodesic segments {x^^^j.^j), 
which are joined at Y and dX. 

Following |Salj . we shall suppose that k is a ternary fraction 

• • • , 

where = or k = 2 for all m, so that k is a point of the Cantor set 
in [0, 1]. More precisely, we say that the trajectory has type k if the 
following is true. 

• If the (m + l)st segment of the trajectory is obtained by re- 
flection and there exists the corresponding refracted ray then 

= 0. 

• if the (m -|- l)st segment of the trajectory is obtained by refrac- 
tion then Km = 2, 

• If the trajectory has only m segments or the mth segment ends 
at dX or at a point of total reflection then either k.^ = or 

The last condition implies that a billiard trajectory may have different 
types K. Roughly speaking, the equality = 2 means that (m + l)st 
segment is obtained by refraction whenever it is possible. 

3.3. Dead-end and grazing trajectories. A billiard trajectory is 
not well-defined if 

• the trajectory hits DM infinitely many times in a finite time, 

• or the angle of incidence or the angle of refraction is equal to 
zero, 

• or the trajectory hits a point in 9Msing. 

Trajectories of the first type are called dead-end, trajectories of the 
second type are said to be grazing, and we call trajectories of the third 
kind singular. Let Oj, Og, and be the sets of starting points of the 
dead-end, grazing and singular trajectories, respectively. Clearly, Oj, 
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Kf, = (2,0,0 



Figure 3. Ray-splitting trajectories 

Og, and Os are conic subsets of T*(X \ dM). Throughout the paper 
we shall suppose that the following assumption holds. 

Assumption 3.1. The conic set Od IJ U has measure zero in the 
cotangent bundle. 

This assumption means the set of starting points of "bad" trajecto- 
ries is of measure zero or, in other words, that the billiard trajectories 
(x^d/, 77), ^* (y, r])) are well-defined for all k, all t > and almost all 
{y,r^)eT*X. 

Remark 3.2. One can easily show that Og [jOs is a set of measure zero 
(see, for instance |Salj . |SVlj or |SV2j ). However, there are reasons 
to believe that Od H S*M may have a positive measure. In |SV1] the 
authors constructed such an example for a similar branching billiard. 

Let Ot be the set of points (|/,r/) G T*(X \ dM) such that all the 
billiard trajectories {x^KiyjV):^i.{y:V)) ^.re well defined for t G [0,T] 
and x^{y, rj) ^ dM. Clearly, Ot are open conic subset of T*(X \ dM). 

= nT>o ' 



Assumption 



3.1 



Ot is a set 



implies that their intersection O^^ 
of full measure in T*X. 

Note that the mapping $^ : (y, r^) 1— (x^, ^* ) defined on Ot is a ho- 
mogeneous canonical transformation in the sense in symplectic geome- 
try for each fixed t G [t, T] and k. It preserves the canonical symplectic 
1-form ^ ■ da; on T*X and the standard measures on T*X and S*M 
(see, for instance, |Sal] or |S V2] ) . 



4. Fourier integral operators 



Further on we shall abbreviate the words 'Fourier integral operator' 
and 'pseudodifferential operator' to FIO and ^DO, respectively. We 
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shall always be assuming that their symbols and amplitudes belong to 
Hormander's classes (see, for instance, |H2l Chapter 18]). Recall 
that the conic support conesuppp of a function p G is defined as 
the closure of the union [jj supppj, where pj are the positively homo- 
geneous functions appearing in the asymptotic expansion p ~ J2jPj- 

4.1. Definition. Let $ : {y,r]) ^ {x* {y , t]) , C,* {y , f])) be a smooth ho- 
mogeneous canonical transformation in T*M defined on an open conic 
set ©($) C T*M, and let V : C^{M,n^/'^) ^ C°°{M,Q^/^) be an op- 
erator with Schwartz kernel V{x,y) (that is, Vu{x) = (V(x, ■),«(■))). 

The operator V is said to be a FIO of order m associated with $ if 
V{x,y) can be written as an oscillatory integral of the form 

(4.1) (27r)-" / e'^^'^'y'My^v) \det^,,{x,y,r])\'/' <,{x,y,r^)dr^ 
Jt*m 

modulo a half-density from C°^(M x M,Q}^'^). Here and p are 
smooth functions on M x T*M satisfying the following conditions. 

(ai) is an arbitrary cut-off function, which is positively homoge- 
neous of degree for large rj, is identically equal to 1 in a small 
neighbourhood of the set {x = x*{y,ri)} and vanishes outside 
another small neighbourhood of the set {x = x*{y,ri)}. 

(3.2) p is an amplitude from a class with conesuppp C ^^($); 

(as) is positively homogeneous in rj of degree 1 with Imip > 0. 

(a4) ip{x,y,r]) = {x - x*) ■ ^* + O {\x - a;*p) as a; -)■ x*. 

(as) det ipxrj{x,y,ri) 7^ for all {x,y,ri) G supp^ and a; = x* is the 
only solution of the equation ipr^{x,y,ri) = on supp?. 

Note that 

• (f.ri{x*,y, f]) = because <I> preserves the 1-form x-d^. Therefore 
the condition (as) is fulfilled whenever det ipx-q{x*,y, ?]) 7^ and 
supp<j is small enough. 



• The right hand side of (4.1) behaves as a half-density with re- 
spect to X and y and, consequently, the corresponding operator 
acts in the space of half-densities; 

Remark 4.1. The above definition of a FIO was introduced in |LSVj 
(see also |SV2[ Chapter 2]). It is equivalent to the traditional one, 
which is given in terms of local real-valued phase functions parametriz- 
ing the Lagrangian manifold 

(4.2) {(y,r/;x,0 eP($) xT*M : {x,0 = ix*iy,v),C{y,v))} 
(see, for example, |H1] or fTr\). 
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Remark 4.2. By |LSVl Theorem 1.8], the Schwartz kernel of a FIO 



can be represented by an integral of the form (4.1 ) with any phase func- 
tion if and cut-off function q satisfying the above conditions. 

Remark 4.3. One can define a FIO using ( 4.1[ ) with an amplitude 



p{x,y,ri) E depending on x instead of p{y,rf). These two defini- 
tions are equivalent. Indeed, since {x — x*)e^'^ = SV^e*"^ with some 
smooth matrix-function B, one can always remove the dependence on 
X by expanding p into Taylor's series at the point x = x* , replacing 
{x — a;*)e*'^ with i? Vj^e*'^ and integrating by parts. In particular, this 
procedure shows that ( |4.1 ) with an x-dependent amplitude p{x, y, t]) de- 



fines an infinitely smooth half-density whenever p = in a conic neigh- 
bourhood of the set {x = x*}. 

One can find all homogeneous terms in the expansion of the am- 
plitude p{y, rf) by analysing asymptotic behaviour of the Fourier trans- 



forms of localizations of the distribution (4.1 ). This implies that p{y, rj) 
is determined modulo a rapidly decreasing function by the FIO and 
the phase function ip. It is not difficult to show that the conic support 
cone supp p does not depend on the choice of ip and is determined only 
by the FIO V itself (see, for instance, jSV2l Section 2.7.4]). We shall 
denote it by cone supp V. 

Remark 4.4. By \SV2\ Corollary 2.4.5], if a phase function satis- 
fies (as), (a4) and the matrix lmip^^{x* ,y,ri) is positive definite then 
det ip xrj{x* ,y,ri) ^ 0. One can deduce from this that the set of phase 
functions ip satisfying (as) -(as) is connected and simply connected. 

4.2. The Keller— Maslov bundle. Let T'2($) be the Z-principal bun- 
dle over !?($) on which the multivalued function axgdet^ ipxr^ix* ,y, if) 
becomes a single valued continuous function of y, rj depending contin- 
uously on ip. The fibre at the point (|/, rj) can be thought of as the 
set of equivalence classes of pairs {ip,a), where is a phase function 
satisfying (a3)-(a5), a is an integer, and the equivalence relation is 
(V?,a) ~ ((^,a) iff 

27r(a - a) - argdetVxr,(a;*,l/,^?) + argdet^(^^.^(x*,?/,?7) G (-7r,7r], 

where the branch of the argument in the right hand side is chosen to 
be continuous along any path in the set of phase functions satisfying 
(a3)-(a5). Then aigdei^ipxri{x* ,y,r]) can be defined as a continuous 
single valued function on the principal bundle Vi{^) determined by 

argdetVx,7(a;*,l/,r7, [{<p,a)\) G (-vr + a, tt + a]. 

Locally real phase functions provide local trivializations of Vj^i^^) de- 
termining the topology on the total space. 
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In the following we will often suppress the argument a)] and 
think of the function aTgdet'^(fxri{x* ,y) as a multivalued function on 

understood as a continuous single valued function on 
Factoring out 4Z C Z one obtains a Z4-principal bundle which we 
denote by Vz^{^). 

The complex line bundle associated with the representation 

is called the Keller-Maslov line bundle. Our definition here is equiva- 
lent to the one given in the literature (see, for instance, |H1] and [Tr] ) 
because if ipxr^{x* ,y,ri) and (pxr,{,x* ^y^rf) are real then a) ~ ('^, a) is 
equivalent to 

a-a = {sgnip^r,{x*,y,r]) - sgn <pr,^{x* ,y,r])) . 

Sections of the Keller-Maslov line bundle can be understood as func- 
tions f{y,ri, a) on V^^{^) satisfying the equivariance condition 

(4.3) f{y,v,a + n)=t-^f{y,7^,a), 

where a = [{(f), a)] denotes the variable in the fibre on which the Z- 
action is defined by [(0, a)] + n = [(0, a + n)]. For the purpose of this 
article we will think of them in this way. 

The situation simplifies when the bundle is topologically triv- 

ial. This is equivalent to the existence a branch of arg(det^ (px'qix*,y, rj)) 
which is continuous on the set D($) . Clearly, such a branch exists 
whenever !)($) is simply connected. 

4.3. The index function 0$. Let C = Ci + iC2 be a symmetric 
n X n-matrix with a nonnegative (in the sense of operator theory) real 
part Ci, and let lie be the orthogonal projection on kerC. We shall 
denote det+ C = det(C -|- He)- Furthermore, we define the function 
arg det+C in such a way that it is continuous with respect to C on the 
set of matrices with a fixed kernel and is equal to zero when 6*2 = 0. 
In particular, if Ci = then arg det+ C = | sgn C2 where sgn C2 is the 
signature of C2 (see |H21 Section 3.4]). 
The following is jLSV| Proposition 2.3]. 

Proposition 4.5. The function 

(4.4) Q>s>{y,i]) = ^ aTgdet'^<^xn{x*,y,r]) 

Zn 

- ^ aTgdet+{(pr,r^{x*,y,r])/i) + ^ rankx*(?/, 77) 
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as a function on does not depend on ip and local coordinates. The 

function 0$ is continuous along any path on which rank x* is constant. 

Definition 4.6. //7 : [a, 6] is a path in let 7 : [a, 6] 

Vzi^) be any continuous lift. 0$(7(6)) — 9$(7(a)) is independent of 
the lift and called the Maslov index ofj. 

The above definition was introduced in |LSV] (the idea goes back to 
|Ar] ) ■ If the path is closed then it coincides with the Maslov index de- 
fined in |H2] via a Cech cohomology class associated with parametriza- 



tions of the Lagrangian manifold (4.2) by families of local real-valued 
phase functions. The index function B$ allows one to extend this no- 
tion to non-closed paths. 

4.4. The principal symbol of a FIO. Choosing ^ with a sufficiently 
small support, we can rewrite (4.1) in the form 

(4.5) (27r)"" [ e^'^(^'^'^)g(y, r]) (det^ ^,,(x, y, r])) ^'^ ^(x, y, r/) dri , 

JT*M 

where q is the section of the Keller-Maslov bundle obtained from the 
amplitude q{x,y,r]) = p(y,r]) e~^^^^^^^^^ v^v{^^y,v)) i^y ^^i^e procedure de- 
scribed in Remark 4.3 Clearly, q belongs to the same class S'^g and 

(4.6) go(y,^?) = po(2/,r/)e-t(^'-s(<^^*'^-(^*'J^'^)), 

where go and pq are the leading homogeneous terms of the amplitudes 
q and p. 

By construction, (det^ y^xr]{x, y, is well defined for x sufficiently 

closed to X* as a continuous function on Vz^{^) satisfying the equiv- 
ariance condition 

(4.7) f{y,v,a + n)=t-f{y,v,a), 

where a is a variable for the fibre of Vz^i^)- Therefore it is a sec- 
tion in the dual of the Keller-Maslov bundle. Since the product of 
q{y,ri) [det^ ipxri{x,y,ri)Y^'^ is single valued, q{y,ri) is a section of the 
Keller-Maslov line bundle. We shall call it a full symbol of the corre- 
sponding FIO. 

The following is \SiV2\ Theorem 2.7.11]. 

Theorem 4.7. Let V be a FIO whose Schwartz kernel is given by 
(4.5). Then the leading homogeneous term go of the amplitude q is 
uniquely determined by the operator V . 



Remark 4.8. By Theorem 4/7, the leading homogeneous term go does 
not depend on the choice of local coordinates and does not change when 
we change the phase function (f in the representation (4.5). 
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Remark 4.9. Theorem 4.7 was proved in |SV1] only for canonical 
transformations associated with billiards. However, the same proof 
works in the general case. 

In the following we will denote the leading homogeneous term go 
of the symbol of a FIO V by cry and think of it as a multivalued 
function on or as a single valued function on Vz^^l^) respectively, 



that satisfies (4.3). Note that the product is single valued by 

construction as 

where po is the leading homogeneous term of the amplitude p from (4.1 ) 
and argdet+(v3r)r)/'^) is evaluated at x = x*{y,ri). 

Example 4.10. //$ is the identical transformation then y, 77) = 

I and the corresponding FIO V is a ipDO (see, for example, (Sh] The- 
orem 19.1]/ // we put arg(det^ = then the principal symbol of 
V (in the sense of the theory of pseudodifferential operators) coincides 
with the leading homogeneous term go- 
lf the bundle is trivial we can globally fix a branch of 

arg(detVx,?(a;,y,r/) 

for a fixed phase function. Note, that in general there will be no pre- 
ferred branch. In case V is a ipDO we shall however suppose that 
arg(det^ (fxri) = 0, so that in this case ay coincides with the traditional 
principal symbol. 

4.5. Symbolic calculus for FIOs. In what follows, in order to avoid 
'boundary effects' when considering compositions of various FIOs and 
ipDOs on M, we shall have to assume that supports of their Schwartz 
kernels are separated from the boundary. Namely, we shall deal the 
following classes of operators. 

• A'q is the class of operators V whose Schwartz kernels V{x,y) 
vanish in a neighbourhood of the set dM x M. 

• Aq is the class of operators V whose Schwartz kernels V{x,y) 
vanish in a neighbourhood of the set M x dM. 

• ^ is the class of operators that can be written in the form 
cl + Aq where Aq E Aq. 

The following results are in principle well-known. However, we need 
explicit formulae for the principal symbols, which are not obvious 
(partly, due to the fact that there are many possible definitions of the 
symbol). Therefore we have included a simple direct proof of Theorem 
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4.11 in Appendix [Aj In the rest of this section b = b(?/, 77, ai, 02) or 
b = b{y,ri,a) will denote a map between the fibres of the Z4-bundles 
that depends only on the involved canonical transformations. A con- 
struction of this map is implicitly contained in the proofs of the state- 
ments but is not needed for our purposes. 

Theorem 4.11. Let Vj be FIOs of order rrij associated with canonical 
transformations where j = 1,2. Assume that either Vi G A'q or 
V2 G Aq. Then the composition V2V1 is a FIO of order nii + m2 asso- 
ciated with the canonical transformation $ = $2 ^ o $1 with principal 
symbol equal to cry2*Vi(?/,77, b) = (Jvi(l/, t], cii) 0"V2($(?/, 77), 02) such that 

cone supp (V2*Vi) C ^cone supp Vi |^ $^^(cone supp V2) 

Recall that the inner product in the space of half-densities L'^{M, fi^^^) 
is invariantly defined, and so are the adjoint operators. Taking Vi = I 
in Theorem |4.11 we obtain 



Corollary 4.12. Let V be a FIO of order m associated with a canonical 
transformation $. IfV& then the adjoint operator V* is a FIO of 
order m associated with the inverse transformation with principal 
symbol equal to (7v*{y,ri,b) = cry($~^(?/, r/), a) such that 

cone supp V* C $(cone supp V) . 



Theorem 4.11 and Corollary 4.12 immediately imply 



Corollary 4.13. Let Vi, V2 be as in Theorem A.ll Assume that either 
Vi, V2 ^ A'q or V2 £ -^0- Then V2V1 is a FIO of order mi+m2 associated 
with the transformation $ = $2 o ^1 with principal symbol equal to 
(^V2Vi{y,V,b) = CTVid/,^, ai)o-y2($i(|/,r7),a2) such that 

cone supp (V2V1) C ^conesuppVi |^ ^( cone supp V2 

Remark 4.14. In simple words, the above formulae for principle sym- 
bol s mean that av^*v^{y,ri) = i'^i cry^(?/, ?]) cry^ ($(?/, 77)) , av*iy,v) = 

i''^ cry ($-1(^,77)) and av^vAv^v) = i''^ (^Viiv^v) (^V2i^iiy,v)) , where 
kj are integers which are uniquely determined by the canonical trans- 
formations. 

It is well known that FIOs can be extended to the space S'{M\dM) 
of distributions with compact supports in M \ dM (see, for instance. 



|Sh] or [E])- Theorem 4.11 and standard results on ipDOs imply that 
a FIO of order m lying in ^0 is bounded from H^{M) to H'^~"^{M). 
Let A,B& Aq be ^/'DOs, and let V he a FIO associated with a 



canonical transformation $. Then, by Corollary 4.13, AVB is a FIO 



SEMICLASSICAL THEORY OF DISCONTINUOUS SYSTEMS 



17 



associated with $ such that cone supp (AVB) is a subset of 

cone supp i? 1^ cone supp y |^ (cone supp A). 

The above inclusion imphes that WF{Vu) C $(WF(m)) for all dis- 
tributions u G £'{M \ DM), where WF (■) denotes the wave front set. 
Roughly speaking, this means that singularities of a distribution are 
moved by the map $ under the action of the associated FIO V . 
The following is a refined version of Egorov's theorem. 

Theorem 4.15. Let Vi and V2 be FIOs of orders mi andm2 associated 
with a canonical transformation $. If A & Aq is a tpDO of order m 
then 

(1) the composition B := V2AV1 is a ipDO of order m + mi + m2 
such that 

cone supp i? C cone supp Vi |^ (cone supp A) |^ cone supp V2; 



(2) aBiy,v,(^) = (^vAy^V,a)(^Ai^iy,v))(^V2iy,V,b) 

(3) z/ Vi = V2 then (TB{y,v) = (^A{^{y,v)) Wv^{y,v)?■ 



Proof. The first two statements are obtained by applying Corollary 4.13 



to the composition AVi and then Theorem |4.11| to the composition of 
this operator with ¥2- 

If Vi = V2 then aB{y,f]) coincides with aA{^{y,f])) Wvi{y,v)\'^ &s 
the principal principal symbol of a FIO, that is, modulo a factor i'^ 
with k G Z4. If y4 is a non-negative self-adjoint operator then so is 
B = V*AVi. It follows that as > 0, which implies that k = 0. Since 
aB continuously depends on A, the same is true for all ipDOs A. □ 

5. The unitary group e^'*^^''^ 

In this section and further on we shall denote 

U{t) := e-'*^''". 

Since A is self-adjoint, the operators U{t) form a strongly continuous 
unitary group in the space of half-densities in L'^{M, 

5.1. Representation by FIOs. In the following theorem de- 
notes the class of operators whose Schwartz kernels are infinitely smooth 
on [0, T] X M X M up to the boundary. 

Theorem 5.1. Let A and B be tpDOs in the space of half- densities on 
M such that A G Aq, B G A'q and cone supp S C Ot- Then, on the 
time interval [0,T], we have AU{t)B = Ekj modulo 
where the sum is finite and U^jit) are parameter- dependent FIOs such 
that 
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(1) each FIO U^jit) is associated with the canonical transformation 

■ (y^v) ^ {xij{y,v),^ij{y,v)), where {y,r]) e conesuppE 
and (a^Kjj'CKj) ^-^ « billiard segment defined in Subsection 3.2: 

(2) each FIOs Ui^j{t) satisfies the equation d^U^jit) — AUi^j{t) = 
modulo 

(3) the FIO Ui^fi{t) associated with the first segment (which starts 
att = 0) satisfies the initial condition f/K,o(0) = /, for all other 
segments Unj{0) = 0; 

(4) the phase functions corresponding to the incoming, reflected and 
refracted trajectories coincide on the set {x G F |J9X}, and so 
do the arguments aj:g{det'^ ipxri); 

(5) the principal symbols cri/^.(^t) o^re (locally) independent oft. 

The theorem is proved using the standard technique, which goes back 
to |Ch] . For a usual elhptic boundary value problem with branching 
billiards, it is discussed in detail in |S V2j . For the operator defined in 
Section [2| a sketch of proof was given in |Salj . 

Remark 5.2. The same arguments show that a similar result holds 
for negative times. More precisely, Theorem 5.1 remains valid for t G 
[— T, 0] under the assumption that conesuppi? C O^j,, where is 
the set of starting points of billiard trajectories going in the reverse 
direction which are well defined on the time interval [— T, 0]. One can 
easily show that [y, rj) G if and only if [y, —rj) G Ot. 

5.2. Principal symbols of the FIOs U^j{t). Following [ 5V2l Sec- 
tion 2.6.3], let us fix branches of arg(det^ 933:^) for the FIOs Uf^j{t) 
assuming that 

• for the FIO U^fiit) associated with the first segment of billiard 
trajectories, arg(detVxr,) I^^q^^^^^ = 0; 

• the branches corresponding to the incoming, refiected and re- 
fracted trajectories coincide on the set {x G Y[JdX}. 

In view of the parts (3) and (4) of Theorem 5.1, these two condi- 
tions can be satisfied. Clearly, they uniquely determine the branch of 
arg(det^ V^a;^) for all the FIOs f/Kj(t) for all times t. It follows that 
the bundles Vz{^i_j) over cone supp B associated with the transforma- 
tions are trivial. This allows us to consider the principal symbols 
an,j(t;y,ri) of FIOs U^jit) as single- valued functions on M_|_ x T*M. 
In particular. Theorem 5.1 5) implies that the principal symbol ^(^t) 



associated with the first segment is identically equal to 1. 

Let Om,T be the conic set of points {y,ri) G Ot such that all the 
billiard trajectories {xl^{y,r]),^l^{y,ri)) experience at most m refiections 
and refractions for t G [0,T]. The sets Cm,r are open, and their union 
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over m contains Ot- Since the intersection S*M f] cone supp B is com- 
pact, the conic support cone supp 5 is covered by a finite collection of 
connected components of Om,T with a sufficiently large m. Assume, for 
the sake of simplicity, that cone supp B lies in one connected component 



of Om,Ti and let UK,j{t) be one of the FIOs introduced in Theorem 5.1 
Since there are no dead-end, grazing or singular trajectories of length 
t G [0, T] originating from Om,T, for each fixed k and j there are two 
possibilities : 

(i) all segments {x^^ .j{y,ri),^j, j{y,ri)) with (y, //) G cone supp i? end 
at the points of T*M\y which do not belong to the set of total 
refiection; 

(ii) all segments {x^^ ,j{y,ri),^j, j{y,ri)) with {y,ri) G cone supp i? end 
at the points of total refiection in T*ML ; 

I -frog 

(iii) all segments (a;^ ,,(?/, r]), ^* ,,■(?/, 77) ) with {y,ri) G cone supp -B end 
at T*MLy . 

In the first case, in order to satisfy the boundary condition, one has 
to add to UK,j(t) two FIOs Uf,gj+i(t) and corresponding to 

the refiected and refracted trajectories, respectively. 

In the second case, the FIO Ui^Qj^i(t) corresponding to the refiected 
trajectories is chosen is such a way that the Schwartz kernel of the sum 
Unjif) + f^Ko,i+i(^) has singularities only at the points {t,x,y) with 
X G y. After that one can satisfy the boundary condition by adding a 
'boundary layer term' whose singularities are also located at the points 
(t, X, y) with X G F (see jSV2t Section 3.3.4] for details). Since A G A!q, 
the boundary layer terms do not appear in the sum representing the 
operator AU{t)B. 

Finally, in the third case, the Dirichlet or Neumann boundary con- 
dition is satisfied by adding only a FIO f/Ko,i+i(^) corresponding to the 
refiected trajectories. 

Let a,,j{t;y,r]), a^^j+i{t;y,r]) and a^^j+i{t]y,r]) be the principal 
symbols of the FIOs Uf^j{t), ?7koJ+i(^) and U^-^j+iit). 

Lemma 5.3. Assume that the trajectory (x^ j{y, 77), ^* j{y, 77) approaches 
Yj-eg from the g+-side and hits Y^eg at the time t*{y,ri) at the point 

{x*,C) ■■= lim 

t^t*{y,ri)-0 " '-' 



Let T± be as in Subsection 3.1, and let f_ = a/|5'+(x*, ^*) — g-{x*,^p)\ 



if (x*,^*) is a point of total reflection. Then 

(5.1) a^gj+i{t*;y,7]) = T^,^j+i{y,r])a^j{t*;y,7]) 

(5.2) a^,j+i{t*;y,r]) = T^,,j+i{y,v)(^Kj{t*;y,v) 
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where 

(i) in the first case r^oj+i = ^^Hf^r ^^u+i ~ 



(ii) m i/ie second case TkoJ+i = 



If the trajectory hits the boundary SXreg then we have (5.1) with 



111 r, 



Ko,j+i = 1 for the Neumann boundary condition, and 
TkoJ+i = — 1 for the Dirichlet boundary condition. 

Clearly, if the geodesies approaches l^eg from the (^--side, then the 
coefficients t^^j+i and t^j^j+i are defined by the formulae which are 
obtained from the above equalities by swapping + and — . 

Remark 5.4. Note that (tkoj+i)^ + (tk^j+i)^ = 1 in the case (i), and 
Ir^oj+il = 1 in the cases (ii), (iii). 



Part (iii) of Lemma 5.3 is a particular case of |SV2l Corollary 3.4.7]. 
The formulae (i) and (ii) can be deduced from |SaH Proposition 3.3], 
which states the same result but for principals symbols defined in a 
different way. However, the proof of Proposition 3.3 in |Salj is very 
sketchy and is not easy to reconstruct. Therefore in Appendix [B] we 
outline a direct proof, which uses the technique developed in |SV2] . 

5.3. The index function of billiard transformations. In this sub- 
section we shall briefly recall some results from [SV2^ Section 1.5] and 
|SV2l Appendix D.6]. Strictly speaking, they were proved in |SV2j only 
for billiards obtained by reflections. However, the same technique of 
matching phase functions is applicable to refracted trajectories, and 
the proofs remain exactly the same. 

Let us denote the index functions of the transformations • by 



Q {t; y , rj) . Theorem 5.1[4) implies that the index functions corre- 



sponding to two consecutive billiard segments coincide at the points of 
reflection and refraction. This allows us to define the index function 
Qi^it^y.rj) associated with the transformation : {y.rj) i— )■ (x^,^*). 

Consider the matrix of the first derivatives {x*^)n{y,r]). Since x^. is 
positively homogeneous in rj of degree zero, rank (x^)^ is not greater 
than n — 1. If rank (a;^)^ < — 1 for some s > then the point 
is said to be a conjugate point of the billiard trajectory x^^{y,ri), and 
the number n — 1 — rank (x^)^ is called its multiplicity. At the points 
of reflection and refraction, rank (x^)^ is the same for the incoming, 
reflected and refracted trajectories. Thus the notions of a conjugate 
point and its multiplicity are well defined for all {y, rj) G Ot and all 

[0,T]. 

The following statement is an immediate corollary of |SV2l Lemma 
1.5.6] and [5V2l Theorem D.6.8]. 
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Proposition 5.5. If{y,ri) G Ot then every billiard trajectory x^^iy^rj), 
t e [0,T], has only finitely many conjugate points. The number of its 
conjugate points counted with their multiplicities is equal to — ©^(T'; y, rj) 



6. Local Weyl asymptotics 

The operator A is self-adjoint and lias compact resolvent and there- 
fore there exists an orthonormal basis in L'^{M, il^/^) such that 

where < Ai < A2 <...—>■ oo. 

Given an L^-bounded operators A, let us define 

The Weyl asymptotic formula for A^(A) implies that 

NaW < \\A\\N{\) < const A" . 
We shall denote 

whenever the limit exists. Formulae giving the value of A(A) in terms of 
other characteristics of the operator A (such as its symbol or Schwartz 
kernel) are usually called local Weyl laws. 

Remark 6.1. Since A(i^) = whenever the operator K is compact, 
Iy{A) depends only on the image of A in the Calkin algebra. 

Clearly, Na{X) is a function with locally bounded variation, whose 

1 /2 

derivative N'j^ is the sum of 5-functions located at the points A^ with 
coefficients {A(t)i,4>j). For every smooth rapidly decreasing function p 
on M we have 

00 

(6.1) ^p(A-Af')(>10,,0,) = p*K{X) = p'*Na{X), 

j=i 

where * denotes the convolution. In many cases it is easier to investi- 
gate the asymptotic behaviour of p * A^^(A). After that, a local Weyl 
law can be obtained by applying a suitable Tauberian theorem. 
li A e Ao and its Schwartz kernel A{x, y) satisfies the condition 

(C) {x, y, 0) WF (A) for all (x, e T*M and y e M 
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then the trace Tr {AU(t)) exists as a distribution in t. Indeed, the 
above condition imphes that AU{t){l + A)"™ is an operator with a 
continuous kernel UA,m(t,x,y) for all sufficiently large positive integers 
m, so that we can define 

TT{AU{t)) := (i-^) J^ UA,Ut,y,y)dy. 

In this case 

(oo 

= m Tr (AUm = im Tr iUit)A)) , 

where p is the Fourier transform of p and J-'j~^x inverse Fourier 

transform. 

Let V G ^0 be a FIO of order zero associated with canonical trans- 
formations $. By |H2l Theorem 8.1.9], the wave front of the Schwartz 
kernel of is a subset of 

{{x, -r]) G T*M X T*M : (x, = Hy, r])} , 

Therefore V satisfies the condition (C). 
Further on 

• <luj{y,vi) = Y 1^*^^ ^'^^ normalised measure on S*M] 

• Fix($) = {(?/,?7) e S*M : ^{y,f]) = {y,fi)} is the set of fixed 
points of a transformation $ lying in S*M. 

The following two results are proved in Appendix [Cj 

Theorem 6.2. // V G a F/0 o/ order zero associated with a 

homogeneous canonical transformations $ then the limit A{V) exists 
and 

(6.3) AiV) = [ z®*(^'^)cry(i/,r/)da;(y,77). 

JFix(<I>) 

Lemma 6.3. // Vj G ore F/Os o/ order zero associated with ho- 
mogeneous canonical transformations $j and let V = V2V1 and $ = 
$2^ o $1. Then 

i®^av = (i®*iavi)(r®*2a^) 
almost everywhere on the set Fix($). 



Theorems 4.11 6.2 and Lemma 6.3 immediately imply 
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Corollary 6.4. // Vj G ore FIOs of order zero associated with 
canonical transformations then the limit A(V2*Vi) exists and coin- 
cides with 

A(V^;V1) = / (^''^^ av, {y, fi) z"®*^ (^'^^) ay, (y, 77) <lu{y, fi) , 

JFix($) 



Applying Theorem 6.2 with $ = J and the Weyl formula for the 



counting function, we obtain the following well known result. 

Corollary 6.5. If A E A is a ipT^O of order zero then the limit A(yl) 
exists and is equal to fg,^j(jA{y,fi)du{y,fi). 



Remark 6.6. A theorem similar to Theorem 6^ was stated and proved 
under a clean intersection condition in |Z2] . Note that the formula for 



A{V) given in |Z2j differs from (6.3), as it does not contain the factor 
i®* . This is due to the fact that the author used an implicitly defined 
notion of "scalar principal symbol". An explicit definition of this object 
would have involved the index function or an analogue. 

7. Classical dynamics of branching billiards 
7.1. Definitions. Let O-r be the conic subsets of the cotangent bundle 



T*{X \ dM) defined at Subsection 3.3 By Assumption 3.1, Ot is an 



open set of full measure for each T > 0. 

If {y^rj) G Ot then all the biUiard trajectories (a^tl?/? ^7)) 
originating from {y,ri) are well defined for t G [0,T] and experience 
only finitely many reflections and refractions. It follows that, for each 
fixed (?/, rf) G Ot and t G [0, T], the set of end points of the trajectories 
(yX^^iy y v) : ^i{y y v)) is finite. Let us denote it by ^^{y,ri). 

Remark 7.1. Note that ^^^{y^rj) is not uniquely defined if the trajec- 
tory hits the boundary at the time t* . For the sake of definiteness, we 
shall be assuming that in this situation ^* (|/, ''?) := ^vm.t^t* ^^^{y , rj) . 

Remark 7.2. Recall that the shifts along billiard trajectories $^ : 
{y^ri) I— !■ (x^,^*) are homogeneous canonical transformations in T*X. 
One can consider the branching billiard system as a family of multi- 
valued canonical transformations $*, mapping {y,ri) G T*X into the 
set <l>\y,v) = [j^Kiy,v)- 

Suppose that a billiard trajectory 77), ^* (y, 77)) is well defined 

and hits the boundary at the times < tl{y,ri) < tgd/; ''?)•• • For 
each t* {y, rj) we have the associated coefficient r^^j+i {y, rf) calculated 



in Lemma 5.3, where j ' + 1 is the order number of the next segment. 
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More precisely, in the notation of Lemma 5.3, t^j+i := t^qj+i if the 
next segment is obtained by reflection, and r^j+i ■= TkiJ+i if the next 
segment is obtained by retraction. 
Let us define 



iiO<t<tliy,r]) 
Ut- {y,v)<t (?/> V) if tl {y,ri) <t. 



In view of Remark |5.4[ 

(7.1) Yl = 1 

for all {y,r]) G Ot and t G [0,T], where the sum is taken over all 
distinct billiard trajectories of 'length' t originating from {y,ri). 

Remark 7.3. We call the number \Ti^(t;y,ri)\'^ the weight of the trajec- 
tory {xf^{y,ri),C,^{y,ri)), s G [0,t]. It can be thought of as the proportion 
of energy transmitted along the billiard trajectory, or the probability for 
a particle to travel along this trajectory. 

If (x,^) G ^^{y,ri), let us denote 



wt4t;y,v) ■■= I Yl ^''''^''''''^r,it;y,v) \ , 



where Tj^it; y, rj) are as above, 9^ are the index functions introduced in 
Subsection 5^, and the sum is taken over all distinct billiard trajecto- 
ries of 'length' t originating from {y,T]) and ending at (x,^). 

In view of Assumption 3A, the following definition makes sense for 
all p G [1, oo] and t > 0. 

Definition 7.4. The classical transfer operators and the diagonal 
transfer operators in the space LP{S*M,du) are defined for times 
t >0 by the equalities 

i^tf){y,v) ■= Y '^Ud'^-^y^v) fi^^O , 

{Eff)iy,f,) := Y wf,d^;y,f,)fix,0, 

where {y, fj) G S*{M \ dM) \ Ot . 

The difference between S'^ and is in the contributions from re- 
combining billiard trajectories, that is, the billiard trajectories such 
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that {xlii) = ^',6) but ^ for some s e (0,t). If 

the set of initial points that admit recombining bilhard trajectories has 
measure zero then and coincide. 

Example 7.5. Let X he the unit 2- dimensional sphere and Y be a 
great circle, splitting X into the union of two hemispheres Xj^ and X_ . 
Let us provide X± with the metrics c± g, where g is the standard met- 
ric on X and c± are positive constants, and consider the Riemannian 
manifold X = X+ |J X_ . In this situation, the billiard trajectories are 
formed by the great semicircles lying either in X+ or X_, whose length 
is equal to 7rc+ and 7rc_., respectively. If m^c+ = m^C- with some 
positive integers m± and t > 47rm+c+ then for every billiard trajectory 
r]), ^* (y, r/)) there exists a distinct trajectory {x^f^,{y,fi),^j,,{y,fi)) 
with the same end point, which is obtained from (a^^d/' ''Z)? Ck(i/5 ''?)) 
replacing 2m_|_ great semicircles in X^ with 2m_ great semicircles in 
X_ or the other way round. If and c__ are rationally independent 
then there are no recombining trajectories. 

7.2. Results. The following theorem reveals the link between the di- 
agonal transfer operators and local Weyl asymptotics. 

Theorem 7.6. Let B,C & Ao be tpDOs of order zero in the space of 
half- densities on M such that cone supp i? IJ cone supp C C Ot. Then, 
for all ipDOs A E A of order zero and all t G [0, T], 

K{CU*{t)AU{t)B) = [ ac{y,f])Ef{aAKy,f])(rB{y,v)du{y,fi). 

J S*M 

Proof. If A is the multiplication by a constant then the theorem follows 



from Corollary |6.5[ Thus we can assume without loss of generality that 
A e .Ao. 



Then, by Theorem 5.1, U{t)B and U{t)C* can be represented as 



finite sums of lOFs U,,j{t)B and ZIk'j' Ut^ijt{t)C*. It follows that 
(7.2) CU*{t)AU{t)B = CU:,^^,{t)AU^,j{t)B, 

Corollaries |4l2| and |4l3| imply that {CU*,j,)* = U^'j'C* and AU^^j{t)B 
are FIOs associated with canonical transformation $* , •, and $1 with 
principal symbols 



c^c/,/ V) (^c{y, V) and aA{^^j{y, v)) (^u^,,{t){y, v) (^siy, v), 

respectively. Clearly, the set of fixed points of the mapping ($^, j/)"^^^, 
consist of {y, rj) G T*M such that 
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Now the required result is obtained by calculating the principal symbols 
of U^jit) and UK'j'it) with the use of Lemma 5.3, applying Corollary 



6.4 and summing up over all segments of the billiard trajectories. □ 



Corollary 7.7. // (y,r/) eOt andO<t<T then 

^Uoi^^y^V) = and ^ w'l^^^){t,y,r]) = 1 . 



Proof. The first equality is an immediate consequence of (7.1). The 
second is proved by applying Theorem 7.6 to A = / and comparing the 
obtained result with Corollary |6.5 □ 



7.3. Properties of the transfer operators. Clearly, 
• and are positivity preserving operators. 



Corollary 7.7 implies that 

• the operators and Ef are continuous in all spaces Lp{S*M, du) 
with p E [l,oo] and their operator norms in these spaces are 
bounded by 1, 

• and are isometries in the space L^{S*M,du). 

Note that the operators form a semigroup, whereas may not 

coincide with 

Definition 7.8. Let be either or Sf. We say that St is ergodic 
if for all f G L'^{S*M, du) 

1-2 



(7.3) 2T-M iEJ){y,f,)dsdt ^ / f{y,f])du{y,fi) 

Jo Jo Js*M 

as T ^ +00 almost everywhere in S*M. 



In view of Corollary 7/7 and Lebesgue's dominated convergence the- 
orem, if St is ergodic then 

(7.4) r f (S J) dsdt ^ ( [ f{y, f,) duj{y, f,)] 1 

Jo Jo \Js*M J 

in all spaces Lp(S'*M, dw), where 1 is the function identically equal to 
one. 

Remark 7.9. Changing the order of integration, one can rewrite the 
condition (7.3) in the following equivalent form 

f{T - s) (H J) {y, fj) ds [ f{y, 77) du{y, f,) . 

Jo JS'M 
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Remark 7.10. The traditional definition of ergodicity assumes that 



(7.5) 



f{y,v) duj{y,r]) 



S*M 



as t +00 almost everywhere. It is easy to see that (7.5) implies 
(7.3) but, generally speaking, the converse is not true. However, if the 
left hand side of (7.5) does converge to a limit for all f (as in the 
von Neumann ergodic theorem) then, by (7.3), the limit coincides with 
fg^^jf{y,fi)dijj{y,fi) and, consequently, the dynamics is ergodic in the 
classical sense. In our scenario this happens when Y = and there 
are no branching trajectories. 

8. Classical ergodicity implies quantum ergodicity 

The purpose of this section is to prove the following theorem which 
is the main resuh of this paper. 



Theorem 8.1. Suppose that Assumption 3.1 is fulfilled and that the 
diagonal dynamics is ergodic. Then quantum ergodicity holds, that 
is, for any ipDO A E A of order zero we have 

N 



hm 



-E 



{A(f)j,(j)j) 



aA{y,r]) duj{y,r]) 







Proof. The proof proceeds in several steps. 

Step 1. We can assume without loss of generality that 







S*M 



simply by subtracting the constant J ^^^^ a A{y,v) duj{y,ri) fiom A. Thus 
it is sufficient to prove that, under the assumption (8.1), 

1 ^ 

limsup — = 0. 



Af-s-oo 



Step 2. If Q is an L^-bouned operator, let us denote 



AAr(Q) 



1 ^ 

-T 



j5 rj/ ! 



so that A((5) = limAr_>oo Ajv((5) whenever the limit exists. By the 
Cauchy-Schwarz inequality, 

N N 

(8.2) _^|(g0^,0^.)| < -Y,\\Q<\>,\\ < iAr,iQ*Q)f\ 
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Let US define At := U{-t)AU{t) and At := T'^ U{-t)AU{t)dt . 
Here and further on integrals of operator-valued functions are under- 
stood in the weak sense. 

Since U{t)(j)j = e"'*-^j''Vi and U{-t) = U*{t) for all t G M, we have 

(^T0j,0j) = {At(j)j,(j)j) = {A(j)j,(f)j) 



for all positive integers j, t eM. and T > 0. Therefore, by (8.2), 

1 ^ / 
limsnp — ^\{A(t>j,(f)j)\ < i Urn sup An {A*rp At) 



N , X 1/2 

\ 

Af-s>oo ^ V Af^-oo 

Thus it is sufficient to show that 

(8.3) limsupA^(i^iT) = 0. 

7V->-oo 

Remark 8.2. Note that, generally speaking, the operators At and At 
are not FIOs and do not belong to Aq. Therefore we cannot directly 
apply Theorem 6.2 or Corollary 6.4 to evaluate the upper limit (8.3). 

Step 3. Clearly, 

(8.4) \\AT(pjf = T-^ [ [ {U{-t)AU{t)(f)j,U{-r)AU{r)^j) dr dt 



JO 



= T-^ [ [ FA,j{t - r) dr dt , 
Jo Jo 

where 

Fa,j{s) := e~'''^y\u{-s)A<f)j,A(f),) = (A*?7(-s)Af/(s)0,-, 0,) . 



Since Faj{—s) = Faj{s) and is real, the integral on the right 

hand side of (8.4) coincides with 

/" f FA,j{t~r)drdt = 2T-^ [ [ FAj{s)dsdt. 

Jo Jo ' Jo Jo 

Thus it follows that 

/•T i-t 

I 2 o rp—2 



(8.5) {A*^AT(pj,(t)j) = \\AT(j)jr = 2T-M / {A* A,(f)j,(j)j) ds dt . 

Jo Jo 

Step 4- Let Op(x) be the operator of multiplication by a real- valued 
function x e C^(M \ dM) such that < x < 1, and let B e Aq he a 
ipDO of order zero. Since ||A*74s|| < ||y4||^ and ||Op(x)|| < 1, we have 

\{A*AscPj,<Pj) - (Op(x)AM,0„0,)| = |(Op(l-x)A*^0i,0,)| 
= |(AM,0„Op(l-x)0,)| < \\Af\\Opil~xW\LHM) 
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where Op(l — x) is the multiphcation by 1 — x, and 

= |(Op(x)AM,(/-i?)0„0,)| < \\Af\\iI-B)(f),\\L^M)- 



These inequahties and (8.5) imply the estimate 

fT ft 



{A*tAt(P,Aj) - 2T"M / {Op{x)A*AsB<f),,<j),)dsdt 

Jo Jo 

< \\Af ||Op(l - x)4>j\\lHm) + \\Af ||(/ - B)^,\\l2^m) 



Now, applying the second inequality (8.2), we see that 



(8.6) 



An{A*tAt) - 2T-'- / {A^{Op{x)A*A,B)dsdt 

Jo Jo 

2\l/2 , II ^112 /A ^rr n\*^T 7-)^^^l/2 



< WAf (A^((Op(l - x)f) + (Aiv((/ - BTil - B))y 



for all iV = 1,2,... 



Step 5. Let us fix an arbitrary e > 0. In view of (8.1), ergodicity of 



the diagonal dynamics implies that there exists T > such that 



2T" 



T rt 



JO 



H (Tyi ds dt 



< e. 



Li(5*A/,dw) 



Let us fix such a positive T and choose the nonnegative function x ^ 
C^(M \ dM) and the ipBO B e Aq of order zero such that 



(a) ll(l-X)lli2(5.M,da.) 

(b) conesuppi? C Or, {ctbI < 1 and ||1 — as] 



L2(5*Af,daj) 



Note that (b) can be satisfied because, by Assumption |3.1 , Ot has full 
measure. 



In view of Corollary 6.5, the limits A((/ — Op(x))^) and A((/ 
B)*{I — B)) exist and are smaller than e"^. Therefore 



[*) the right hand side of (8.6) is estimated by 2£:||A|p for all suf- 
ficiently large A^. 



By Theorem 7.6, the limit A{Op{x) A* A^B) also exits and is equal 



to 



S*M 
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The Lebesgue dominated convergence theorem imphes that 

(8.9) hm 2T-^ [ [ (A^(0p(x)AM,5) ds dt 
Jo Jo 

= 2T-^ [ [ {A{Op{x)A*AsB)dsdt. 
Jo Jo 

Substituting ( |8.8[ ), integrating over s and t, and taking into account 
(8.7) and (a), (b), we see that the absolute value of (8.9) is smaller 
than esup|crA|. Consequently, 

(**) the integral in the left hand side of (8.6) is estimated by 
esup Icr^l for all sufficiently large A^. 



Applying the estimates (*) and (**) to (8.6), we obtain 
limsupAAr(y4^AT) < 26 + esup IctaI . 



Since e can be chosen arbitrarily small, this implies (8.3). □ 

Appendix A. Proof of Theorem 14.111 

In this and next sections, if / = (/i, . . . , /„) is a vector-function of 
n-dimensional variable 6 = {6i, . . . ,6n), we denote by fe the n x n- 
matrix function with entries {fi)ej, where j enumerates elements of the 
ith row. 

Let 

and 

(A.2) $ := o $1 : {y,7]) ^ (x* (y , v) . Civ , v)) ■ 

Note that 

(A.3) (C«)-zf - (zf )-Cf = (Cf ) - (zfrct = , 
(A.4) (C«)-zW - (zW)-Ci^) = iC^Yz^^ - izfre^ = I 

(A.5) {zl^m^^^ = = 0' 

(A.6) {zl,'YC^'^ = V and (zf fC^^^ = ^ 

for all {y,ri) and (x,^) in any local coordinates because the transfor- 
mations $j preserves the 2-form dz A d( and the 1-form ( ■ dz. 
The proof proceeds in several steps. 
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A.l. Part 1. Assume that <l>i(?/o,''?o) = '^'2(3^05^0) := (-20, Co)- Then 
there exists a local coordinate system z = {zi, . . . , Zn) in a neighbour- 
hood of zq such that det(i,^\yo,r]Q) ^ and det C^^^(a;o, ^0) 7^ 0. 

Indeed, let z be arbitrary coordinates in a neighbourhood of zq. One 
can easily show that under a change of coordinates z ^ z the matrices 
C^^'' and CP'' transform in the following way 



(A.7) 



I ~(2) 
\z=zW ' 



where C-^-* are symmetric matrices with entries 



(1) 



/ J Sm 

m=l 



(2) 



=z(i) 



EC 

m=l 



(2) 



dzi dzk 



2(2) 



(see, for instance, |SV2l Section 2.3]). Clearly, one can choose coordi- 
nates z in such a way that 



■^z)\z = zW{yQ,r)o) 



=2(2)(a;o,Co) 



and C*^^^(?/o, r^o) = C^'^\xq,^q) = cl , where c is an arbitrary real 
constant. Then ( A.7[ ) turn into 



In view of (|A^, Ci^^ and Cf' map the kernels of the matrices Zrj 

~(2) 

and z^ into the orthogonal complements of their ranges. This implies 
that the matrices in the right hand sides of the equalities (A. 8) are 
non-degenerate for all sufficiently large c . 



:(2) 



=(1) 



A. 2. Part 2. Let {yQ,rio), (xo,^o) and {zqXo) be as in Part 1, and let 

z be an arbitrary local coordinate system such that det ^^^^(yo, ^0) 7^ 
(2) 

and detQ (xo,^o) 7^ 0. 

Assume first that cone supp Vi lies in a sufficiently small conic neigh- 
bourhood Oi of the point {yQ.rjo) such that det C)7^^(?/, 77) 7^ for all 
{y^r]) G Cl and detcf^(x,0 ^ for all G $i(Ci) . If 

V'^'H^,!/,^) = {z-z^'\y,^))-C^^\y,^), 
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then, by (A.5), 



^Pz'rj = Crj^^ and (p^^^ = C^^'' • Therefore the phase functions (p^^^ satisfy 
the conditions (a3)-(a5) of Subsection 4.1 for all {y,ri) G Oi , (x,^) G 
$i((9i) and z sufficiently closed to zq. 

Since WF{Viu) C for all distributions u, we may assume 

without loss of generality that conesupp V2 C $i((9i) . Then, in view 
of the above, the Schwartz kernels Vi{z,y) and V2{z,x) of the FIO Vj 
can be represented by oscillatory integrals 



,(2) 



.(2) 



(A.IO) 



(2vr)- 
(2vr) 



T*M 



M'H^,y,v)p^^y^^) |detCi')| / <;^{z,y,v)dr^, 
M''(^'^'^^p,{x,0 |detCf I'''' ^2(^,x,0de 



of the form (4.1) with the phase functions (A. 9). The Schwartz kernel 
of the composition V2V1 coincides with 



(A.ll) 




Jilj{x,^,z,y, 



'^^K^, z, y, 7]) pi(y, 7]) P2{x, drj dz d^ 




|^|-ng.^(.,|,|C,.,,,,)^(^^ |r/|e, z, y, v) Piiy, V) P2ix, m dr] dz d^ , 
where the integrals are taken over T*M x M x T*M, 

i;{x,^,z,y,v) = iz-z^'\y,r]))-C^'\y,r])-iz-z^'\x,0)-C^'\x,0 
and 

b{x,^,z,y,r]) 



= (27r)-2"|detCf)(y,r^)|'/' detCf y, 77) <^2(^, x, • 

Now we are going to apply the stationary phase method to the in- 
tegral with respect to the variables z and ^, considering |?7| as a large 
parameter. A rigorous justification of the stationary phase formula 
for non- convergent integrals of this type can be found, for instance, in 
jSV2l Appendix C]. 

The equations ip^ = and ipz = are equivalent to 

(A.12) z = z^'\x,0 and C^'\x , = C^'\y , v) 

respectively. Since (^^^(a^c^o) = C^^^(z/o,'7o) and det Cf ^(x p, ^p) ^ 0, in 
a neighbourhood of (xq, yo, rjo), the second equation (A.12 ) has a unique 



1/2 
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^-solution ^{x,y,ri) such that ^(xq, yo, ^70) = ^o- Thus the stationary 
point is (z,^) = {z^'^\x , ^) , ^) . It is unique and non-degenerate because 



(A.13) 



C 



(2)^ 



T 



VV 



By the stationary phase formula, the integral (A. 11) coincides mod- 
ulo a smooth function with 

^ -1 



{2n) 



detCf^(a;,0 p{x,y,ri) dr] . 



(A.14) 
where 

(A.15) (fix, y, 77) := ^, z^'^\x, y, rj) 



z^^\x,i)-z^'\y,^)].C'''Ky.v) 



and p is an amplitude of class with the leading homogeneous 

term 

(27r)2" z^^\x, i),y, r]) pi{y, rj)p2{x, C) 

such that 

conesuppp C {{x,y,ri) : {y,ri) G conesupppi, (x,^) G conesuppp2} 



(we have used the fact that the signature of the Hessian (A.13) is equal 
to zero). 

Clearly, i{x*{y,ri),y,ri) = C{y,v) and z^'^\x*,C) = z^^\y,v)- Thus 
(f{x*,y, rf) = 0. Since 

ilj^{x,i,z^'^\x,i),y,ri) = tjj^{x,i,z^'^\x,i),y,T]) = 

for all X, y, i], we also have 

ip,ix,y,r]) = Mx,iz^'\x,i),y,r]) = {z^^\x,i)f C^'\x,i) . 



Now the second equality (A. 6) implies that (px{x,y,ri) = ^{x,y,ri). 
Substituting x = x*, we obtain 

(A.16) ^Ax*,y,v) = iz^^\x*,C)fc^'\x*,C) = C. 

Similarly, 

^vi^,y,ri) = tlj^{x,i,z^^\x,i),y,r]) 

= V,{iz-z^'\y,r^))-C^'\y,r^))\ 
This equality and ( |A.5 ) imply that 

^vi^^y^v) = - z^^\y,r])) ■ C^/\y,v) 



\z=z(^) {x,i) 
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and, consequently, 

Differentiating the identity C^'^\x,i) = C^^\y,Ti), we obtain 
From the above two equahties it follows that 



Vx,j{x*,y,r]) 



(2) _ .(2) 



^™ 2/. 



Cx 1 Cjj 



(1) 



where 4 = A''{y-,V)-, Ci' = Cv^'{y,v)^ and z^^' , z}f' , (x''' , Of' are 

evaluated at ( x*,^ *). 

In view of (|A.3l) and (|A.4l), 



.(1) 



(2) ^(2) A-i) A2) 



.(2) 



Z. 



(cf)"cf) 



(C 



(2)xT 



-1 



(C 



(2)nT 



Consequently, 

(A.17) ¥..,(x*,y,r^) = {iCl'\x*,nry\if\y,v) 



and 



(A.18) |det^.,(x*,2/,77)|^/' = \detC^'\y,v)\'^' detCf (x*,^ 



-1/2 



Since detC^^'^ 7^ 0, the above equality and (A. 16) imply that the 
phase function (A. 15) satisfies the conditions of Subsection 4.1 This 
shows that (A. 14) defines the Schwartz kernel of a FIO associated with 
the canonical transformation $. Applying the procedure described in 
Remark 4^, we can remove the dependence on x and rewrite it in the 
form 

(A.19) (27r)-" [ e'^^^'y^^^piy,v) \det cpx,{x* ,y,v)\'^' ^(x, y, r/) dr/ 



where ^ is a cut-off function satisfying the conditions of Subsection |4.1 
and p{y, rj) is an amplitude of class S'^^"^^ such that 

conesuppp C {{y,ri) G conesupppi : {x*,C) ^ conesuppp2}- 

Clearly, the leading homogeneous term po of the amplitude p is given 
by the formula 

(A.20) 
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where pj^ are the leading homogeneous terms of the amphtudes pj. 

A. 3. Part 3. Consider now general FIOs Vj associated with the canon- 
ical transformations $j. Splitting Vj into sums of FIOs with simply 
connected conic supports, we see that it is sufficient to prove the the- 
orem assuming that the bundles V^i^j) are topologically trivial and 
the Schwartz kernels of Vj are given by oscillatory integrals Ij of the 



form (4.5) with phase functions (pj and full symbols qj. 

If the conic supports of qj are sufficiently small then, choosing suit- 
able local coordinates and transforming ipj into the phase functions 



(A. 9), we can rewrite the corresponding oscillatory integrals in the 
form (A. 10) with pj = i^^qj, where kj is an integer determined by the 



choice of branch of arg(det {cpj)xri)- In this case, by Part 2, the com- 



position V2V1 is a FIO given by the oscillatory integral ( |A.19[ ) with 

phg 



the local phase function (|A.15[) and an amphtude p G S'^^J''^^ with the 



leading homogeneous term 

Let if be an arbitrary global phase function associated with the trans- 
formation $. Since !)($) = T'($i) is simply connected, the bundle 
T)x{^) is also trivial. Let us fix a continuous branch of arg(det^(<^ 



Transforming the phase function Lp given by (A. 15) into Lp, we see that 



(A. 19) coincides with an oscillatory integral of the form 

(2vr)-" / e'^^^^y'^\-'q{y,r]) (det^ ^.,(x, y, r/))'/' ?(x, r^) dr/ , 



where q G S'^^"^'^ is another amplitude with the same leading homo- 
geneous term i^i~'^2 q^ ^ijj^ rj) ^2,0 {^{v, v)) ^ is the integer such that 
^ = arg(det^v9a;^) turns into arg(det^(^2:^) under continuous transfor- 



mation of the phase functions (p ^ ip (see Remark 4.8). 

Thus have proved that, for Vj with small conic supports, the com- 
position V2V1 is a FIOs of order mi + 1712 with principal symbol 

such that 

cone supp {V2V1) C ^cone supp Vi $^^(cone supp V2] 

Obviously, the integer ki — k2 — k is uniquely defined by the choice 
of branches of &Tg{det'^{(pj)xri) and arg(det^((^i.^). Therefore, using a 
partition of unity on T*M, we see that the same result holds for all 
FIOs Vj. Since the principal symbols are defined modulo a factor i"^ 
with an integer m, this completes the proof. □ 
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Appendix B. Sketch of proof of Lemma |5.3[ i) and (ii 



Let the FIOs corresponding to the incoming, reflected and refracted 



trajectories are given by the oscillatory integrals (4.5) with phase func- 
tions v?, if^ , and symbols g, and q~ respectively. The first two 
are standard oscillatory integrals defined in Section 111 In the case (i) , 



the third is also a standard oscillatory integral. In the case (ii), it is a 



boundary layer oscillatory integral given by the same expression (4.5) 
but with a complex-valued phase function satisfying the conditions of 
jSV2t Section 2.6.4]. 

Substituting the sum of the integrals into the boundary conditions 
and equating to zero the sum of leading terms at t = t* and x = x* , 
we obtain the following equations, 

(B.l) ip{t*,x*,y,r]) = ip+{t*,x*,y,r]) = ip-^H* ,x* ,y,ri) , 

(B.2) (w^jCiv, + M«o,i+lC^^+)L=t*,x=a;* = J+1 ^V'" ) L=t* , x=x* 

and 

(B.3) {u^j a+v? + 

where := (det^ "ipxrjY^^ and denote the inward gf-t-normal deriva- 
tives. 

The condition (a4) implies that 

Similarly, in the case (i), ^~^~\^^^» = r_ . In the case (ii), by 
[SVa (2.6.23)], we have ,1,. = if. . 



From the equalities (B.l), [5V2l (2.5.3^)] and f5V2l (2.6.25)] it fol- 



lows that at the point t = t*, x = x* 
-r+ det if^r^ = r+ det (p^^ 

(cf. [SV2l (2.6.14)]). Consequently, = Vlt=t.,x=x. and 

I -1 /t , /t (T? - in thp 

d^_ 



r_ detip^^ in the case (i), 
i f_ det (p~^ in the case (ii) . 



\/t^/tI d^ in the case (i), 

v=x* j 

where (r+/if_)^^^ is a continuous branch of the square root. 
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In view of the above equalities, the equations (B.2), (B.3) imply that 



in the case (i), and 



u 



-r+ u^j + r+ = -if_ (r+/? f_)^/^ 



in the case (ii). Solving these equations with respect to MkoJ+i 5 '"kij+i 
we obtain the required formulae for the coefficients Troj+i j ''"kiJ+i • 



Appendix C. Proofs of Theorem 16.21 and Lemma 16.31 



C.l. Proof of Theorem 6.2 , Without loss of generality we shall be 
assuming that cone supp V lies in a sufficiently small neighbourhood of 
a fixed point (?/o, r?o) eT*{M\ dM). 

Let t G (—5, 5) with a sufficiently small 5. If 5 is smaller than the 
geodesic distance from the support of the Schwartz kernel of V to the 



boundary dM then, in view of Theorems 5.1, Corollary 4.13 and Re- 



mark 5.2, VU{t) is a FIO associated with the canonical transformation 
$y := $ o $* with principal symbol o'vu(t){y^ v) = <^v{^^{y, v))^ where 

$* : {y,r]) ^ {x\y , r]) , , r])) 

is the shift along geodesies in M \ dM. Denote 

$(l/,r/) := ix*{y,r]),Ciy,v)) , Kiv^v) ■= {z\y , v) , C iv , v)) ■ 

Assume that 6 and cone supp V are small enough. Then the union 
Ut6(-5 5) "^onesupp V^(t) is also small and, for t G (—6,6), the Schwartz 
kernel of VU{t) can be represented (modulo smoothing operators) by 
an oscillatory integral 



(C.l) (27r)- 



Jip{t;x,y,ri) 



1 /2 

P{t]y,v) |detC*(?/,r?)| <;{t;x,y,r])dr] 



with a phase function given by the equality 

(C.2) ^{t,x,y,r]) = {x - z\y,7])) ■ C\y,7]) 

in a local coordinate system such that det C^(?/, r]) 7^ 0, an amplitude p 
with small conic support, and a cut-off function q satisfying the condi- 



tions of Subsection 431 (see Appendix [A|) . 
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Lemma C.l. Let T,s{^) be the set of points {y,fj) G S*M such that 
^viy^v) = (y^v) some time t* = t*{y,'fj) G (—6,6). If the Schwartz 
kernel ofVU{t) is given by (C.l) and p G C^{—6,6) then 

(C.3) T-^.mTrivum 



= (27r)-"A"-^ / e^^* p(t*) po(t^ y, v) dy dfj + o(A" 

as A — 7- +00, where t* = t*{y,fi) and po is the leading homogeneous 
term of the amplitude p. 

Proof. Clearly, = x* {x^{y,ri),^^{y,ri)). Therefore 
Since $ preserves the 1-form ^ ■ dx, we have 



{x;{x\e)fc = {x;{x\e)fcix\e) = o 

and, consequently, ■ J^^* = ^* ■ ^x*. By Euler's identity for homo- 
geneous functions, C,* ■ ^x* = ^* ■ h^{x^,C,^) = where h{x,^) = 
\/g{x,^). Now, differentiating (C.2), we see that 



(C.4) iptit;x,y,ri) = -^g(x\e) + {x - z\y,7])) ■ -C{y,v) 7^ 

for all {t, X, y, rf) G supp <j provided that supp <; is small enough. 
Let 

1 1 /2 

(C.5) p{t;y,r]) = f){t)p{t-y,ri)\detCl{y,r])\ q{t,y,y,ri) . 

Then 

(/)(t)Tr(\/f/(t))) 

= (27r)-"-i /" [ e*('^(*™)+^*)p(t;?/,r/)d?/dr/dt +0(A-°°). 
J Jt*m 

Changing variables rj = Xrfj, where r G [0, +oo) and fj are coordinates 
on the cosphere S*M, we obtain 

(C.6) T-_l,m^riVU{t)Opix))) 

= {2ny-^\" JJ [ e'^^^'-^'^y'^^p{t;y,\rfi)dydf]drdt + Oi\-'^), 
with 



S*M 



^{t]r,y,ri) = rip{t;y,y,7]) + t . 
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Now we are going to apply the stationary phase formula with respect 
to t and r (see |SV2t Appendix C] for justification of this procedure). 



In view of (C.4) 



on suppp. Thus all stationary points of lying in suppp are non- 
degenerate. They are given by the equations 

(C.7) ip{t;y,y,f]) = and npt{t; y , y , f]) = 1 . 



By (C.4), the first equation either does not have any solutions, or has 
a unique solution t*{y,fi) smoothly depending on {y,fi). Here we have 
used the implicit function theorem and assumed that S is small enough 
so that t*{y,fi) is defined on suppp. 

In the first case ( |C.6 ) is a rapidly decreasing function of A and 



z^iUiV) 7^ y for {t',y,v) ^ suppp, which implies that $y(y,?7) ^ 
{y,ri) for all {y,ri) G suppp and t G supp p. 

In the second case, by the stationary phase formula, 

(C.8) T~_l,{p{t)TT{VUm 

= (27r)-"A"-^ [ e'^'*r*p{t*;y,Xr*fi)dydfi + o{r-^) 

Js*M 

as A — 7- +00, where r* = r*{y,ri) := \(pt{t*]y,y,fj)\~'^. One can easily 



show that the right hand side of (C.8) coincides with 

(C.9) (27r)-"A"-i / e'^'*r*p{t*;y,Xr*ri)dydfi + o{X''-'), 

Jn 

where 



n = {{y,f])e S*M : t;{y, fi) = t;{y, 77) = 0} 

is the set of stationary points of the function t*. 

Differentiating the identity (fit*, y, y,fi) = and taking into account 



(C.4), we see that f2 consists of the points {y,ri) such that 
^^{t\y,y,fi) = {y- z'\y,f]))-C*{y,v) = 

and 

ify{t\y,y,fi) = {I-z'y{y,fi)fC'\y,fi) + {y-z'\y,fi))-Cl\y,fi) = 0. 

The first equation implies that y = z^*{y,rj). Since $y preserves the 
1-form ^-dx, the vector-function Zy{y,r]))'^(^*{y,r]) in the second equa- 
tion is identically equal to fj. Therefore the second equation yields 

C\y,v) = V- 
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Thus we see that {y,fi) G if and only if f/) = {y.fj). From 

here and (C.4) it follows that r*{y,fi) = 1 for all {y,fi) G fl. Recall 
that 



(C.IO) 



for every C°° -function the set of zeros of infinite 
order has full measure in the set of all its zeros. 



In particular, this implies that V^(C**(y, — v) = C^iv^v) — 1 = 



on a set of full measure in fi. Now, substituting (C.5) into (C.9) and 
removing lower order terms of p, we obtain the required result. □ 

Let Op(x) be the operator of multiplication by an arbitrary func- 
tion X e C^{M \ dM) such that V = Op(x) V = FOp(x). Then 
ELo^"K:Kn = V and, consequently, iVy(A) = ELo ^"^ ^v^v„(A) , 
where Vm := 1{V + i"^Op{x)). Note that V;^Vm are linear combina- 



tions of FIOs lying in Aq. Applying Lemma C.l to these FIOs and 



taking into account (6.2), we see that 



p * K^yJX) = m Tr {V:,V^Um = OiX-') , A ^ +00 , 

whenever suppp C (—5,5) for a sufficiently small 6. Assume, in addi- 
tion, that the function p is even and nonnegative, and p(0) = 1. Then, 
since the functions Nv*v are nondecreasing, standard Tauberian the- 

* 777, * m t-" 

orems imply that 

\Nv^vJX) - P * Nv,vJX)\ = 0(A"-i), A^+oo, 

(see, for instance, |Sa2t Theorem 1.3]). 

Clearly, the same estimate holds for the function Ny- Applying (6.2) 



and integrating (C.3), we obtain 



(C.ll) NviX) = p*NviX) 



JT4Jp(t)Tr(Vf/(t)))d/i 



= (27r)-" p^-' [ e^^**p(r) po(t^ y, V) dy df] dp + o(A") . 
If £ > and JlsA^) ■= {iy,v) e ^s{^) : \t*{y,v)\ > e} then 

e'^'*pit^)poit^;y,v)dydfidp = 0(A"-i) . 



n-l 
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Since J^si'^) \ (U>o ^sA'^) ) = ^ oi'^) = Fix($) , letting e ^ we see 
that the right hand side of (C.ll) coincides with 



'0 



' po{0;y,fj)dydfjdfi + o(A") 

Fix(<l>) 



(2vr) 



n 



In view of (C.IO) 



X' 



Po{0;y,fi)dydf] + o(A") . 

Fix(<I>) 

on a set of full measure in 



Fix($), where ip is the phase function (C.2). Now (4.8) implies that 
Po(0;y,fj) = i^'^^y''^^av{y,fi) ^ set of full measure, and (6.3) follows 
from the Weyl asymptotic formula for the counting function A^(A). 



6.3[ Let if^^^ and be arbitrary functions 



C.2. Proof of Lemma 

associated (in the sense of Subsection 4.11) with the transformations 



(A.l) and (A. 2). The condition (a4) and (CIO) imply that 

(2) 



.(1) 



z^i' = and ^r,r,\^= 



vv 



r-qri j I 







(C.12) 

on a set of full measure in Fix($). 

Let Vj and V be arbitrary FIOs given by the integrals (4.1) with 
the phase functions ip^^^ and and let poj and po be the leading 



homogeneous terms of corresponding amplitudes. In view of (4.8) and 
( |C.12| ), we have 

(C.13) i^^ay = po and ay, ^ = Po,ip^ 

on a set of full measure in Fix($). In particular, this shows that the 
restrictions of po and po,i Po,2 to this set do not depend on the choice 
of the phase functions. 

Assume now that the FIOs Vj have sufficiently small conic supports, 
the phase functions (p^^^ are given by (A. 9), and V = V2V1. Then, as 



was shown in Subsection A. 2, V is represented by the integral (A. 19) 



with the phase function (A.15) and an amplitude with leading homo- 



From here and (C.13) it follows 



geneous term pifl{y, r]) p2,o ($(2/, v))- 

that i®*cry = ■j®*iz~®*2 cTyjOv^ on a set of full measure in Fix($). 
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